Skip to content

Fix Unity media stream lifetime leaks and add latest-frame video coalescing#198

Open
xianshijing-lk wants to merge 6 commits intomainfrom
sxian/CLT-1/fix_audio_video_streams_leaks
Open

Fix Unity media stream lifetime leaks and add latest-frame video coalescing#198
xianshijing-lk wants to merge 6 commits intomainfrom
sxian/CLT-1/fix_audio_video_streams_leaks

Conversation

@xianshijing-lk
Copy link
Copy Markdown
Contributor

@xianshijing-lk xianshijing-lk commented Mar 20, 2026

This PR significantly improves audio and video stream stability, particularly for iOS devices, by implementing proper pre-buffering, background/foreground handling, resource lifecycle management, and memory safety improvements.


🎵 Audio Stream Improvements (AudioStream.cs)

Pre-buffering Strategy

  • Unified 200ms ring buffer for all platforms (removed platform-specific sizing)
  • Smart 30ms pre-buffering: Wait for 30ms of data before starting playback to prevent underruns
  • Automatic underrun detection and recovery: When buffer runs low (< 50% of requested samples), immediately output silence and wait for 30ms refill before resuming
  • Result: Eliminates choppy audio during network jitter while maintaining low latency (~30-50ms in practice)

Background/Foreground Handling

  • Clear ring buffer and reset priming state when app returns from background
  • Prevents playing stale audio data after resume
  • Ensures fresh 30ms buffer before resuming playback
  • Fixes: Audio glitches and delays when resuming iOS apps from background

Memory Safety

  • Fixed memory leak: Added using statements for AudioFrame and resampled frames
  • Proper disposal ordering: Remove event delegates before disposing resources
  • Added null checks in audio read callback to prevent crashes after disposal

🎤 Microphone Source Improvements (MicrophoneSource.cs)

iOS Audio Session Handling

  • Proper pause handling: Stop microphone when app goes to background to release AVAudioSession
  • Smart restart logic: Poll for microphone device availability (up to 2s timeout) instead of arbitrary delays
  • Fixes: FigCaptureSourceRemote errors (-17281) and ArgumentNullException during background/foreground transitions

Component Lifecycle Management

  • Use DestroyImmediate() to remove existing AudioSource/AudioProbe before adding new ones
  • Fixes: Race conditions during quick stop/start cycles where Unity's deferred Destroy() caused component duplication
  • Added null safety checks for GameObject validity before accessing components

Error Handling

  • Graceful error handling with try-catch around Microphone.Start()
  • Timeout-based polling (2s) for microphone readiness instead of indefinite WaitUntil()
  • Re-check microphone authorization after background (permissions can change)
  • Clear error logging for all failure cases

🎥 Video Stream Improvements (VideoStream.cs)

Frame Management

  • Latest-frame-wins coalescing: Keep only the newest frame when Rust delivers bursts
  • Separate pending/active frame buffers: Prevents race conditions between frame production (FFI) and consumption (Unity update)
  • Lock-protected frame swapping: Thread-safe handoff between main thread and rendering

Resource Lifecycle

  • Proper disposal of pending frames when stopping stream
  • Immediate frame disposal when stream is not playing (prevents native memory growth)
  • Fixed disposal ordering to prevent late callbacks from accessing disposed state
  • Only allocate VideoFrame objects when FrameReceived event has listeners

🔧 Supporting Infrastructure

RingBuffer (RingBuffer.cs)

  • Added Clear() method to reset buffer state (used for background/foreground transitions)
  • Added SkipRead() method for advanced buffer control

RtcAudioSource (RtcAudioSource.cs)

  • Fixed async audio capture safety: Each captured frame now gets its own NativeArray buffer
  • Why: Native encoder consumes frames asynchronously after request.Send() returns; reusing a single buffer caused race conditions
  • Track pending frames with unique async IDs and release buffers only after callback completes
  • Added comprehensive logging for debugging audio capture pipeline
  • Validation for sample rate/channel mismatches and malformed frames

AudioResampler & FFIClient

  • Improved memory management and disposal patterns
  • Better error handling and logging

iOS Build Support (IosLinkOrderDiagnostics.cs)

  • New post-build processor to fix iOS link order issues
  • Ensures liblivekit_ffi.a links before libiPhone-lib.a to prevent Opus/CELT conflicts
  • Strips conflicting CELT object files from libiPhone-lib.a

Copy link
Copy Markdown
Contributor

@cloudwebrtc cloudwebrtc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@xianshijing-lk xianshijing-lk force-pushed the sxian/CLT-1/fix_audio_video_streams_leaks branch from 77607e1 to 7b94b74 Compare March 27, 2026 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants